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Abstract 

We derive a general result about commuting actions on certain objects in braided rigid 
monoidal categories. This enables us to define an action of the Brauer algebra on the tensor 
space y®*^ which commutes with the action of the orthosymplectic Lie superalgebra spo{V) 
and the orthosymplectic Lie color algebra 5po(V,l3). We use the Brauer algebra action to 
compute maximal vectors in T/®*^ and to decompose T/®*^ into a direct sum of submodules 
T'^ . We compute the characters of the modules T'^ , give a combinatorial description of these 
characters in terms of tableaux, and model the decomposition of T/®*^ into the submodules 
T'^ with a Robinson-Schensted-Knuth type insertion scheme. 
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Summary of results 

In this paper we show that there are orthosymplectic Lie superalgebra and orthosym- 
plectic Lie color algebra analogues of the results developed by Berele and Regev [BR] and 
Sergeev [Se] for general linear Lie superalgebras. Our work corresponds in the superalge- 
bra and color algebra setting to what Brauer did in [Br] by extending to orthogonal and 
symplectic groups Schur's classical results for general linear groups. 

In his thesis [Shi] and a subsequent paper [Sh2] Schur proved that the action of the 
symmetric group Sk on tensor space T/®*^ by place permutations determines the centralizer 
of the action of the general linear group GL(V) on T/®*^. This result, often called Schur-Weyl 
duality, relates in a very fundamental way the representation theory and combinatorics of 
the groups Sk and GL{V). The orthogonal group 0{V) also acts on T/®*^, and Brauer [Br] 
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constructed an algebra, now referred to as the Brauer algebra, which commutes with the 
0(y)-action on tensor space. When V is even-dimensional, the symplectic group Sp{V) 
also has an action on T/®*^, and its centralizer can be described using the Brauer algebra. 

Berele and Regev [BR] and Sergeev [Se] have shown that the action of the symmetric 
group Sk on T/®*^ by "graded" place permutations determines the centralizer of the general 
linear Lie superalgebra when V is Z2-graded, and they have exploited this action to study 
certain modules for the superalgebra and their characters. Fischman and Montgomery [FM] 
have generalized the work of [BR] and [Se] to cotriangular Hopf algebras which arise from 
enveloping algebras of general linear Lie color algebras. 

Our intent is to establish an orthosymplectic version of this theory. In particular, we 

(i) show there is a Brauer algebra action on T/®*^ which commutes with the orthosymplectic 
Lie color algebra spo(y, (the orthosymplectic Lie superalgebra spo(y)-action is just 
a special case); 

(ii) use the Brauer algebra to construct a family of maximal vectors for spo(y, [3) in the 
tensor space (Theorem 3.9); 

(iii) obtain from the Brauer algebra action a direct sum decomposition of T/®*^ into spo{V, j3)- 
submodules naturally indexed by partitions A; 

(iv) use the character theory of the Brauer algebra to compute the characters of the spo(y, [3)- 
modules and to give a combinatorial description of these characters in terms of 
tableaux; 

(v) develop an insertion scheme for the tableaux which models the decomposition of the 
tensor space into the modules . 

Remarks on the results m this paper 

(a) To our knowledge, orthosymplectic Lie color algebras were first introduced in [B], which 
discusses the Brauer algebra action, but does not prove that this action commutes with 
5po(V, 13). The notion of an orthosymplectic Lie color algebra allows us to give a uniform 
proof that the Brauer algebra action on tensor space commutes with the action of the 
orthogonal group (Lie algebra), the symplectic group (Lie algebra), the orthosymplectic 
Lie superalgebra as well as more general group graded orthosymplectic algebras. As 
we discuss in Section 1, orthosymplectic Lie color algebras have a natural root space 
decomposition and a triangular decomposition which are exactly analogous to the Lie 
superalgebra case. The derivation of the maximal vectors which we give in Section 3 
extends the work in [BBL] where maximal vectors are computed for classical case of 
the orthogonal and symplectic Lie algebras. 

(b) The action of the Brauer algebra arises naturally from the structure of the category 
of finite dimensional modules for Lie color algebras. As we show in Section 2, braided 
monoidal categories provide a convenient framework for developing the results on com- 
muting actions. Our arguments in Section 2 will apply to give a centralizing action 
of an algebra (in general it may be not be a Brauer algebra) on tensor space for any 
kind of "Lie like" algebra or quantum group for which the category of finite dimen- 
sional modules has a braided monoidal structure and a special isomorphism between 
V and V* . Thus for example, one can modify our Section 2 with very minor changes 
to obtain an action of the Birman-Wenzl-Murakami algebra centralizing the appropri- 
ate quantum analogue oi spoiV, [3). The idea is essentially the same as that used by 
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Reshetikhin-Turaev [RT], in particular, see for example [CP, Theorem 5.3.8]. It shares 
similarities with the methods that Fischman and Montgomery [FM] apply to get their 
symmetric group action, except that we have chosen to work with the structure on the 
category of modules rather than with the Hopf algebra structure as they do. 

Berele and Regev [BR] give a very interesting combinatorial description of the characters 
of the 0[(m/n)-modules which appear in the tensor space T/®*^ = Vo © Vi, dimVo = 
m, and dimVi = n) by describing them as hybrid Schur functions involving tableaux 
which have a column-strict part and a row-strict part. In a similar fashion we give a 
combinatorial description of the character of the module T'^ as a hybrid symplectic- 
ordinary Schur function given by tableaux which have a symplectic part and a row-strict 
part. The symplectic part is a symplectic tableau of the kind introduced by King (see 
for example, [KE]) to index basis elements for irreducible representations of symplectic 
groups. 

The modules T'^ are the same as the ones considered by Bars and Balantekin in [BBl- 
BB2]. They give Jacobi-Trudi type character formulas, but they do not derive the 
tableaux description that we give here. Bars and Balentekin seem to indicate that the 
modules T'^ are irreducible, but this is not clear to us, either from their work or from 
ours. In fact, R.C. King in personal communication has told us that he has found 
explicit examples of T'^ which are not irreducible. 

There has been other work, notably ([FJ], [CK], [L1-L2]), which describes how to index 
representations of the Lie superalgebra 5po(V) by partitions, but none of these papers 
has given an interpretation for their characters in terms of tableaux. The main ingredi- 
ent in developing the tableaux description is the identity in Theorem 4.24 (h). A very 
similar version of this identity (Theorem 4.24 (i)) appears in [CK]. This identity could 
be used in combination with the work of Sundaram [Su2] to give another combinatorial 
interpretation for these characters. See [Ki2] for a survey of the use of tableaux in the 
study of representations of Lie superalgebras. 

There is an extensive literature of papers by Bernstein and Leites [BL], [Lel-Le2], Kac 
[K1-K2], van der Jeugt, Hughes, King, and Thierry-Mieg [JHKT1-JHKT3], Penkov 
and Serganova [PS1-PS3], [P], Kac and Wakimoto [KW], and others which studies 
representations of Lie superalgebras using Lie theoretic and geometric methods. These 
approaches also yield character formulas, the most general of which is the Weyl-Kac 
character formula. We have not made any effort to understand our character formulas 
in this other setting, although the formulas must be equal in many cases. Even for 
the superalgebra Ql(m/n), the connection between the results of [BR] and [Se], the 
Sergeev-Pragacz character formula, and the Weyl-Kac character formula needs to be 
better understood. King and others have done some work in this direction, (see [Ki2]). 
The relationship between the centralizer approach to the representation theory of Lie 
superalgebras and the approach via Kac modules and typical weights also needs to be 
better explained. 

In our work we have not included proofs of the analogue of Schur- Weyl duality, i.e. 
we have not shown that the actions of the Brauer algebra and the Lie color algebra 
5p (V, l3) each generate the full centralizer of the action of the other. Let us make just 
a few remarks on this point. 

(i) In the general linear-symmetric group case of [BR], [Se], and [FM], one can get away 
with proving only half of the duality and using the semisimplicity of the group algebra 



Tensor Representations for spo{V, (3) 



5 



of the symmetric group CSk to obtain the other half for free. This is not possible 
in the orthosymplectic-Brauer algebra case since the Brauer algebra is not necessarily 
semisimple. 

(ii) The usual trick for proving that general linear lie algebra Qi{V) generates the full 
centralizer of the symmetric group action uses the idempotent X^dgs^ (which cor- 
responds to the trivial S'fc-character) to construct a projection map onto the Qi{V)- 
invariants. Unfortunately, this is not available in the Brauer algebra case since the 
Brauer algebra does not have a one-dimensional module which affords invariants. 

(iii) We have succeeded in establishing various parts of both halves of the duality in our 
orthosymplectic-Brauer algebra setting but have chosen not to include these results in 
this paper. Optimally what one would like is a proof of the duality which handles all 
the cases simultaneously rather than arguing separately for the orthogonal group, then 
the symplectic group, then the orthosymplectic Lie superalgebras, then the quantum 
orthogonal group, etc. 



Open problems 

(1) The relationship between the centralizer approach to the representation theory of Lie 
superalgebras and the approach using Kac modules and typical-atypical weights needs 
to be better understood. To our knowledge this is only partially done even for the 
0[(m/n) case [JHKTl-3], and it is not known how to verify directly that the Weyl- 
Kac character formula for typical representations and the Pragacz-Sergeev character 
formula for 0[(m/n)-irreducible modules in tensor space are equal. Here we have not 
made any attempt to relate our results to the Weyl-Kac formula, although this should 
be done sometime in the future. 

(2) In determining the characters of the modules T'^ , we have shown that they are equal 
to polynomials sc^(a;i, x^^ , . . . , Xr, , yi, y^^ , . . . , y,, , yj^ , 1) which appear as the co- 
efficients of the Schur functions s^(zi, . . . , z^+s) in the identity 



11 (i ZtZj) r + s -f-fr /I \/i -1 \ 



1 <z'<j <r + s 

~ ^ ^ , x-^ ,...,Xr,Xj, ,yi,y^ ,...,t/s,t/j , l)s^(zi , . . . , z^+s). 

(0.1) 

There are two classical identities of Littlewood [Li] and Weyl [We] for the characters 
spfj_ of the symplectic group Sp(2r) and the characters so^ of the orthogonal groups 
S0(2s+l): 

,M<f " ''''^n;=in:=i(i-U)(i--r^^) 



(0.2) 



n (1 - ^'^j') 11(1 + -'i) n 11(1 + y^-'i)^^ + % '^j) 

l<z<j<s j = l j = li = l 

= ^so^,{yi,y'^^,. ..,y,,y~'^,l)s^{zi,. . .,z,). 



(0.3) 
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where is the conjugate of the partition When the orthogonal portion is zero, 
i.e. s = and the variable 1 is not present, identity (0.1) gives the classical result in 
(0.2), and when the symplectic part is zero, i.e. r = 0, identity (0.1) reduces to (0.3). 
Is there a combinatorial interpretation for the functions sc^ which expresses sc^ as 
a hybrid object built from symplectic and orthogonal characters? Our combinatorial 
description of sc^ is as a mixed symplectic-general linear character rather than as a 
hybrid symplectic-orthogonal character. 

(3) Find a general Schur-Weyl duality result, see remark (g) above. 
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1. Lie color algebras and spo{V, (]) 



Lie color algebras 

Let K denote a field of characteristic zero (this assumption could be relaxed, but for 
convenience we stay in characteristic 0). Let G be a finite abelian group with identity Iq- 
A symmetnc htcharacter on G is a map j3:G x G ^ k* into the multiplicative group of the 
field such that 

(1) p{ah,c) = p{a,c)p{h,c), 

(2) I3{a, he) = I3{a, b)l3{a, c), and 

(3) /3(a, b)(]{b, a) = 1, for all a,beG. 

Taking a = in the first relation and & = in the second shows that i3(Ig,c) = 1 = 
l3(a, la) for all a, c ^ G. 

A K-vector space V is G-graded if it is the direct sum V = ®aeG °f subspaces 
indexed by the elements of G. If t; G Va for some a ^ G, then v is homogeneous of degree a. 

Assume /3 is a fixed symmetric bicharacter on a group G. A Lie color algebra (g, G, /3) 
is a G-graded vector space g = ®„gg. Qa with a K-bilinear bracket [ , ]: g x g ^ g such that 

(1) [ga,0i] C gaj, for all a,& e G, 

(2) [x, y] = -l3(b, a)[y, x], for x e Qa,y ^ Qh, and 

(3) [x, [y, z]] = [[x, y],z] + l3(b, a)[y, [x, z]], for x e Qa,y ^ Qh, and all z £ g. 
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When the group is the cyclic group G = {±1} = {(—1)" | a = 0, 1} order 2 and 
1)" , (— 1)') = (—1)"', then g is a Lie superalgebra. In this case it is customary to 

regard the grading as by the additive group of the field Z2 = {0, 1}, = 0o © 0i, with 

I3(a,h) = (-1)"' for a, he I-j- 

Remark 1.1. Since, for a Lie color algebra, the bicharacter /3 is symmetric, l3(a~^,a) = 
(](a, a)"^ = (]{a, a) = ±1. Thus g = g(o) © g(i), where 

0(0) = 0« 0(1) = 0«- 

a,l3(a,a) = l a,/3(a,a) = — 1 

One shows easily that l3(ah, ah) = /3(a, a)j3{h, h), which implies the map a i-^ j3{a, a) G {±1} 
is a homomorphism. It follows that the multiplication in g = g^o) ® 0(i) is Z2-graded, but 
still g may not be a Lie superalgebra. 

The category of finite dimensional modules for a Lie color algebra 

Let g be a Lie color algebra. A Q-module is G-graded vector space V = ®agg Va with 
a g-action q®V^V, x ® v i-^ xv such that 

(1) if X e Qa and v eVb, then xv £ Vab, 

(2) [x, y]v = x(yv) — j3{h, a)y(xv), for all x ^ Qa , y ^ Qh and all v ^ V . 

A Q-module morphism from a g-module M to a g-module # is a K-linear map 4>: M ^ N 
such that 

(1) <j){Ma) C Na for all a EG, and 

(2) (l)(xm) = x(l)(m) for all a; G g and all m G M. 

If M and N are g-modules, then the tensor product M ® N is a g-module with 

(1) G-grading given by (M (g) N)^ = ©„j=, M„ (g) and 

(2) g-action defined by 

x(m (g) n) = xm ® n + j3{h, a)(m (g) xn), (1-2) 

for all X E Qa, m E Mi,, and n E N. 
If f: M ^ U and # ^ 1/ are g-module morphisms, then 

f®g: M ®N — > U ®V 

m® n I — T f(m) (g g(n) 

is a g-module morphism. 

The braiding morphisms are the g-module isomorphisms defined by 

Rm,n- M®N — > N®M 

m <E) n I — r j3{h, a)n <E) m 

for all m e Ma, n e Ni,. 

The trivial Q-module is the one-dimensional K-vector space 1 = k with grading 1 = li^ 
and g-action defined hy xv = for all a; G g and all t; G 1. The dual of a g-module M is the 
vector space M* = IIomK(M, k) with 
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(1) G-grading given by (M*)„ = {/ £ M* | /(Mj) = if & 7^ a"!}, and 

(2) 0-action given by 

{xf){m) = -(]{b,a)f{xm), (1.4) 

for all « e 0a, / e (M*)i,, and m £ M. 

It follows easily by direct verification that the category of finite dimensional g-modules 
with the above constructions satisfies the definitions and axioms of a braided rigid monoidal 
category (see [CP] §5.2B). Although we phrase our results in this section and the next in the 
language of braided monoidal categories, the reader need not know anything about braided 
monoidal categories beyond the above standard constructions of modules. To summarize, 
we have 

Proposition 1.5. Let (g, G, [3) he a Lie color algebra. Then with the above constructions 
the category of Unite dimensional Q-modules is a braided strict rigid monoidal category. 

The general linear Lie color algebra qI(V, [3) 

Let G be a finite abelian group with symmetric bicharacter [3. Assume V = ®agg Va 
is a G-graded K-vector space. Let Qi{V, [3) = End(T/) denote the K-vector space of K-linear 
maps from V io V with the G-grading assigned by 

0[(y, = {xe End(y) I xVi C Vai for all h E G] 

and with the bracket 

[x, y] = xy- I3{b, a)yx, 

for all X e 0[(y,/3)„,t/ £ 0[(V",/3)j. Then 0[(y,/3) = 0„gG 0t(^' with this bracket is a 
Lie color algebra, the so-called general linear Lie color algebra. 

The orihosympleciic Lie color algebra 5po(V, [3) 

A [3 -skew- symmetric bilinear form is a K-bilinear map [,):V x V ^ k such that 

(1) the form (, ) on V is nondegenerate, 

(2) (K,T4) = 0ifa^&-i,and 

(3) {v, w) = —l3(b, a){w, v), for all t; £ "14, w £ Vj. 
For each a E G, define 

spo(V, I3)a = e 0[(V^, I3)a I {xu, v) + l3(b, a){u, xv) = 0, for all m £ Vj and v £ V}. 

Then the orihosympleciic Lie color algebra is the Lie color subalgebra 

Sp0(V,P) = ^Sp0(V,P)a 

aeG 

of the Lie color algebra qI(V,I3). 
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In a similar fashion to Remark 1.1 we define 

V(o)= Va and 1/(1) = Va. 

a, /3(a,a) = l a,/3(a,a) = — 1 

The form (,) when restricted to V(o) is skew-symmetric and to V(i) is symmetric. In this 
way y is a Z2-graded vector space with a supersymmetric form. Necessarily dim V(o) is 
even and so throughout this paper we fix 

dim V(o) = m = 2r, and n = dim V(i) = n = 2s or 2s + 1. 

Remark 1.6. If V(i) = (0) and j3{a,h) = 1 for all a,h E G, then spo{V, j3) is just the 
symplectic Lie algebra sp(V(o)) with a G-grading. On the other hand, if V(o) = (0) and [3 is 
such that 

(i) I3{h, a) = —1 for all a, h with j3{a, a) = j3{h, h) = —1, and 

(ii) I3{h, a) = 1 for all a, h such that j3{a, a) = 1 and h) = —1, 

then a) = 1, for all a, h such that /3(a, a) = 1 = &), and spo(y, [3) is the orthogonal 
Lie algebra so(V(i)) with a G-grading. In this way the color algebra approach allows us to 
treat simultaneously the orthogonal and symplectic Lie algebras, the orthosymplectic Lie 
superalgebra (when G = Z2), and all the other orthosymplectic color algebras as well. 

Remark 1. 7, It is convenient in what follows to adopt the convention that (3{u, x) = j3{a, h) 
and (3{u, v) = /3(a, h) whenever m G Va, G Vj, and x G gl(V, l3)i, are nonzero. When notation 
such as l3(u, x) or v) is used, it is tacitly assumed that the elements are homogeneous. 

We assume that the form (, ) (or the field k) is such that there exist homogeneous bases 

Bo = {ti,tl,t2,tl, . . . ,tr,t*}, 

Bi = {ui,ul,U2,U2, . . .,Us,u*,(us+i)}, and (1.8) 
5 = 5oU5i, 

of V(o)) ^(1)) ^iid V , respectively, such that 

(v, w) = {v* , w*) = 0, and {v,w*) = —l3(w* ,v){w* ,v) = Sy^,, 

for all v,w E {^1,^2, ■ ■ - jir, ui, U2, ■ ■ ■ , Ug, (ms_|_i)}. It is to be understood that Ms+i occurs 
only when n = 2s -|- 1 and in that case m*_|_i = Ms+i. We extend the definition of * so that 
{t*)* = ti and (m*)* = Uj for all 1 < i < r and all 1 < i < s. Note if t; £ 5 and v £ Va, 
then V* G Vq-i . 

The matrix of the form ( , ) relative to the basis B is the matrix Fb = (-Fj_j')j_j'g_B with 
F}, },! = {h,h'). More explicitly, 

Ft,r, = 1> ^t-.t. = -1> Fu„u'^ = 1, = 1, (1.9a) 

for i = 1, . . . , r, and j = 1, 2, . . . , s, (s -|- 1), and F}, },t = for all other pairs h, h' G B. The 
inverse of the matrix Fb is the matrix Fg^ = [F^y) with 
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for i = 1, . . . , r, and j = 1,2, . . . , s, (s + 1), and F^^, = for all other pairs b, b' E B. 

Roots and root vectors m 5po(V, l3) 

The choice of the basis 5 of 1/ in (1.8) affords a realization of the elements of qI(V, [3) 
and spo(y, [3) as matrices. The matrix units E^^^, defined by Ey u,y = 6yu,v for v,w,y E B, 
determine a homogeneous basis oi q[{V, [3) with E^^^ G qK^, P)ab-'^ whenever v E Va and 
w e Vj. 

For X G q[{V,13) to belong to spo(y, /3) the relation [xv,w) + j3{v,x)(v,xw) = must 
hold, and that translates to the matrix equation 

v^x^Fbw + j3{v, x)v^Fbxw = 0, 
where "t" denotes the usual matrix transpose. When v,w E B, that equation reduces to 

{x'^)v,w*Fw*,w -\- l3{v,x)Fy „-Xy- yj = Q, or equivalently to 

W* ,V* Ekj* ^KJ -\- l3(^V , X^Fy* ^yXy ^rfjj 0, 

after replacing v with v* . Using the fact that F^* = —(3{v,v* )Fy^y^, we see that 

Fw,W'Xvj',V' = -I3{V,V)I3{W,W)I3{V* ,x)Fy^y*Xy^yj 

must hold for all v,w E B. Therefore, whenever v,w E {ti, . . . ,tr, ui, . . . ,Ug, then 
x^t = —(3{v, v)j3{w, w)j3{v* , x)xy^u,. For such v, w, it follows that Ey u,—l3(b, b)j3{a, b)Eu,t yt 
belongs to 5po(V, l3)ab-i if f G Va and w G Vj. In particular, the elements 

hy = Ey^y - Ey.^y,, V ^ {t I , . . . , t r , U I , . . . , U g} (I'lO) 

belong to 5po(V, /3)io , and they span the space f) of diagonal matrices in 5po(V, 13). Now 
g = g", where g" = G g | [h, x] = a{li)x for all h £ f)}. 

aGf)* 

The set A = {a £ f)* | g" 7^ (0)} is the set of roots of g = spo{V, /3) relative to f). We define 

Aa = {a e A I g" n ga 7^ (0)}, for each a EG, and 
Ao= U A„, Ai= U A„. 

a, /3(a,a) = l a,/3(a,a) = — 1 

Let {ft,} be the dual basis in f)* given by 

iv{hw) = ^v,w, for all v,w E {ti, ■ ■ ■ ,tr,Ui, ■ ■ ■,Us}, 
and set £1,5+1 = 0- Since Ey^y, , Ey,tyt G g[(V^, /3)ai-i if £ Ki, w G Vj, i", w G 5, the elements 
- l3(w, w)l3(v, w)Eyj*^y* eg'""'"-, v,w E {ti, ■ ■ ■ ,tr,ui, ■ ■ ■ ,Us, 

where v ^ w if v,w E . . . , Ug}, 
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' eg' 



V,W ^ {ti, ■ ■ ■ ,tr,Ui, ■ ■ ■,Us}, 

where v ^ w if v,w ^ {ui, . . . , 



'w ,v 



Us}, 



along with the elements hy in (1.10) form a homogeneous basis of g = 5po(V,l3). Thus, 
dim(g") = 1 for all a £ A. 

As a shorthand set fj- = fj^ and 6j = e^^, for 1 < i < r and 1 < i < s. Then the roots 
for 5p (V, l3) have the following expressions in terms of the fj's and 6j's: 

(i) When n = 2s+l, 

Ao = {±(ei ± ej), ±2ei, ±{6k ± 6^), ±6k\l<i^j<r, l<k^l<s} 
Ai = {±ei, ±(ei ±6j)\l<i<r, l<j<s}. 

(ii) When n = 2s, 

Ao = {±{ei ± ej), ±2ei, ±{6^ ± 6^) \ 1 < i ^ j < r, 1 < k ^ £ < s} 

Ai = {±(ei ±6j)\l<i<r, 1 < i < s}, 

which are exactly the same as the corresponding sets of even and odd roots for the simple 
Lie superalgebra 5po(V) which appears in the classical Lie superalgebra theory see [K2]. 

If the simple roots are selected to be 

tti = - fi+i, 1 < i < r - 1, ar+j = 6j - Sj+i, f < j < s - 1, 



then all the simple roots except belong to Aq. Every root a G A is a nonnegative or 
nonpositive integral combination of the simple roots. Accordingly we write a > or a < 0. 
Define elements Xi G g"', 1 < i < r + s, to be the basis elements in (1.11) given by 







't,,t,+i — /3(^i,^i+i)-E't*_|_j,t*, 

kr.ui + /3(^r, Ml)-E'ti*,t*, 
^Uj,Uj^l + , Mj-|-l)£'t,*_|_j,tl* , 

Eus.Us + i + /3(Ws, Ms + l)-E'ti*_|_j,ti*, 

Us-l,U s 1 



1 < i < r - 1, 



l<i<s-l, 
if n = 2s + 1, 
if n = 2s. 



(1.12) 



Then 

g = g_ef)eg+, where g_ = ^g", and g+ = ^ g' 



(1.13) 



a<0 a>0 



The subalgebra g_|_ in (1.13) is generated by the root vectors Xi, I < i < r + s. 



2. The Brauer algebra action on tensor space 



The unfolding map 



A 2k -one- factor is a graph with one row of 2A; vertices and k edges such that each vertex 
is incident to precisely one edge. We draw one-factors so that the edges travel above the 
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row of vertices and denote the set of 2A;-one-factors by 02k- We assume that the vertices 
are numbered 1 to 2k from left to right and often represent a one-factor / G 02k as a 
sequence of pairs / = ((£i, ri), (£2, J'2), . . . , iik,rk)), where r,- e {1, • • • , 2k} give the left 
vertex and right vertex respectively of each edge. As an example, the sequence of pairs 
((1, 4)(2, 7)(3, 5)(6, 8)) represents the 8-one-factor 




12345678 



A k-dtagram is a graph with two rows of k vertices each, one above the other, and 
k edges such that each vertex is incident to precisely one edge. We draw fc-diagrams so 
that the edges remain inside the rectangle formed by the vertices and denote the set of all 
fc-diagrams by Vk- We number the top vertices left to right by 1,2, ... ,k and the bottom 
vertices left to right by V ,2' , . . . ,k' . 

The unfolding map 

u:Vk^02k (2.1) 

converts a fc-diagram into a 2A;-one-factor by relabeling and repositioning the vertices so 
that the sequence (1', 2', . . . , k' , k, (k — 1), . . . , 2, 1) becomes (1,2,..., 2k). As an example, 
«((1, 4')(2, l')(3, 5)(4, 6')(6, 5')(2', 3')) = ((1, 11)(2, 3)(4, 12)(5, 7)(6, 9)(8, 10)), 



1 2 3 4 5 6 




This unfolding map is a combinatorial realization of the isomorphism Endg(T/®*^) ^y®2fc-jB^ 
see Remark 2.13 below. 



The Brauer algebra 

Let K be a field of characteristic and assume rj ^ k. The product of two fc-diagrams 
di and d2 is obtained by placing di above d2 and identifying the vertices in the bottom row 
of di with the corresponding vertices in the top row of d2. The resulting graph contains k 
paths and some number c of closed loops. If d is the fc-diagram with the edges that are the 
paths in this graph but with the closed loops removed, then the product did2 is given by 
did2 = ri'''d. For example, if 






then 
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The Brauer algebra Bk(ri) is the K-span of the fc-diagrams. The K-linear extension of 
the diagram multiplication makes Bk(ri) into an associative K-algebra with identity given 
by the diagram 



By convention Bo(ri) = = k. 

The diagrams in Bk(ri) which have all their edges connecting top vertices to bottom 
vertices form a symmetric group Sk ■ The elements 




1 < i < A; — 1, generate the Brauer algebra Bk(ri). The following theorem is "oft-quoted", 
(see for example [N]), but we are not aware of a proof in the literature. In the interests of 
space we shall continue the tradition and not include a proof here either. 

Theorem 2.2. The Brauer algebra Bk(ri) has a presentation as an algebra by generators 
si,S2, ■ ■ Sfc_i, ei, 62, . . . , Ck-i and relations 



2 _ 1 2 _ _ _ 



l< i < k -I, 



S{Sj — SjS{^ S{6j — 6jS{^ ^i^j — ^j^ii \^ jI'^^i 

SiSi+iSi = Si+iSiSi+i, eiei+iei = ei, Ci+iCiCi+i = Ci+i, l<i<k-2, 
SiCi+iCi = Si+iCi, Ci+iCiSi+i = Ci+iSi, l<i<k-2. 



Some facts about braided rigid monotdal categories 

Let C be a K-linear braided rigid monoidal category whose identity object we denote by 
1. The defining property of 1 is that there are natural isomorphisms V®\ = V = \®V 
for all y G C For simplicity we assume that C is strict. Although everything we do in this 
section works in complete generality for any K-linear braided rigid monoidal category that 
is strict, the reader who prefers not to work with general categorical language can just note 
that for the category of finite dimensional modules for a Lie color algebra g, we have the 
following: 

(2.3) Let V be an object in C. The space of invariants in V is the K-vector space defined 

by 

= Homc(l,y). 

(2.4) Every object V ^ C has a dual, i.e. there is an object V* in C and morphisms 

pry : 1 ^ V" (g) , and evy-V* ®V 1. 



The dual of V is unique up to isomorphism. 
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(2.5) If M and N are any two objects in C, then 

(M ®Ny = N* (g)M*; 
in fact, this isomorphism can be explicitly seen using the canonical maps 



and 



(2.6) There is an isomorphism of K-vector spaces 



Home {M,P®N*) Home {M ®N,P) 

1-^ (f) ' 



where (f) is the morphism defined by 



The inverse map is 



Home (M ®N,P) Home iM,P®N*) 

(f) 1-^ 



where ip is given by 



(2.7) If U and V are any two objects in C, then there is an isomorphism 

Ru,v-U ®V ®U. 

(2.8) If V is an object in C, then the isomorphisms 

R, = id®('-i) ® (-Rv,v) ® id®(*^-'-i) 

satisfy the properties 

RiRj = RjRi, if > 2, 

RiRi + lRi = Ri + lRiRi + 1- 



(2.9) 



T/je braid group action on T/®*^ 

The relations in (2.9) imply that the Ri's afford a representation of the braid group on V^'' , 
(see for example, [CP, §15. 2A]). Note that since Ryy is an isomorphism in Home(V^®^, V^^), 
so is —Rvy. The only reason that we are adjusting by a negative sign is so that the Ri 
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will be the images of the Brauer algebra generators in the map ^ which will be discussed 
in Theorem 2.16. 

If TT is a permutation in the symmetric group define 

where tt = Sj-^ • • • Si^ is a reduced expression of tt (a reduced expression for tt is a product 
of simple refiections Si = {i, i + 1) with p as small as possible). Since the relations in (2.9) 
hold, the map G Homc(V^®*^, V^®*^) is independent of the choice of reduced expression 
(see for example, [Bo, Prop. 5, Chap. IV, §1, no. 5]). If / G O'zk is a 2A;-one-factor, say 
/ = {{li, ri){l2, r'z) ■ ■ - {tk, fk)) with £i < £2 < ■ ■ ■ < tk, let ttj^ be the permutation in the 
symmetric group S2k given by 

^_/l 2 ••• k k+l k + 2 ••• 2A;\^^ 

Then define 

Rf=R^j, for/e02fc, and R^ = Ru(d), for deVk, (2.10) 
where w.'Dk 02k is the unfolding map of (2.1). 

The maps $ and ^ 

Let V be an object of C such that V = V* . Fix an isomorphism 

F:V^V* 

in IIomc(V^, V*). From (2.5) there is a natural isomorphism (y®*^)* ^ (^y*^®*^^ p'j.^ ^nd 
evk be the compositions 

evfc: (T/*)®*^ Cx) T/®*^^(T/®*^)* Cx) T/®*^"^"!. 

Now define a map 

$: 02k — > Homc(l, V"®2«^) = (y®2«^)'^ 
/ ^ 

by letting be the composition 

$^ : 1 P^y»*= id^'^CF^-i)®" yf^2k^yt^2k^ ^2.11) 

Similarly, define a map 

— y Homc(y®^y®*^) 
d ^ *d 

by setting equal to the composite map 

ij.^H*" ^«2J. ^ y^k .d^-T^F^m^-" y^k ^ ^ (2.12) 

id®^g)evi^g,fc -|_ ^ y(dk 
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The map $ associates an invariant $/ in (y®*^)*- to each 2A;-one-factor /. The map ^ 
allows us to define an action of each fc-diagram d on the vector space T/®*^ by letting d act 
by the homomorphism "^d- 

Remark 2.13. The maps $ and ^ are related. Indeed, is the image of (id®*^ Cg) 
F'^^) o e Homc(l, V^^ (g) {V'^^Y) under the isomorphism Homc(l, V^^ (g) (y®*^)*) ^ 

Homc(V"®^ y®*^) which is given by setting M = 1 and # = _P = V"®*^ in (2.6). 

The commuting action of the Brauer algebra on the 5po{V, j3)-module T/®*^ 

Let G be a finite abelian group, [3 a symmetric bicharacter on G, a G-graded vector 
space with a nondegenerate /3-skew-symmetric bilinear form (,) and g = spo(y, /3). There 
is a 0-module isomorphism 

F: V > V* 

V I — > (v,-). 

In this special case, we will compute the map ^ from (2.12) explicitly. 

Let B = {vi, . . . , iijv} be a homogeneous basis of V, i.e. for each I < i < N , Vi G Vc, 
for some Cj- G G. Let {v^ , . . . , v^} be the dual basis in V* . Assume 

Fb = (Fij)i<ij<N, where {vi,Vj)=Fij, (2-14) 
is the matrix of the form (, ) with respect to the basis B. Sometimes we will write Fy^^y^ 
instead of Fi j as we have done in the previous section. Let F~^ = {F~j) i<i,j<N be the 
inverse of the matrix Fb ■ Then 

N N 

F{vi) = Y,Fijvi , and F-^v^) = Y,F-lvi. (2.15) 

i=i i=i 

Let c? be a fc-diagram. Label the top vertices (left to right) with a sequence a = 
(ai , 02, . . . , flfc) of basis elements Oi G B and the bottom vertices (left to right) with a 
sequence h= (&i , &2, • • • , &fc) of basis elements hi ^ B. Assign a weight to each edge and each 
crossing of this labeled fc-diagram according to the following: 

(1) If a horizontal edge (a, a') on the top has a to the left of a', assign the weight Fa^a' 
to it. 

(2) If a horizontal edge (b, b') on the bottom has b to the left of &', weight it by F^^,. 

(3) Weight each vertical edge (a,b) by 6a^h (Kronecker delta), 

(4) Weight each crossing by — /3(i'i , £2), where £1 is the label of a vertex adjacent to 
the first edge, and £2 is a vertex adjacent to the second edge in the crossing. Of the 
four vertices adjacent to the two edges that cross, £1 and £2 should be chosen to be 
the last two vertices (in order) when counting off the vertices in a counterclockwise 
fashion beginning from the bottom left corner of the diagram. 

The weight of the labeled fc-diagram, which we denote da^h, is the product of the weights 
over all the edges and crossings. For example, the weight of the following labeled 7-diagram 
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is the product (-1)^-^3,3-^1, 2-F2_2-F2,4'^l, 2'5l,3l52,4/3(f2, f3)/3(fi, f3)/3(f4, fi)^/3(f4, ^4) =0. We 
have adopted an abbreviated notation in displaying the weights of the edges so as not to 
clutter the picture. 

Theorem 2.16. Let G he a Unite group and let [3 he a symmetric hicharacter on G. 
Assume V is a G-graded vector space with a j3-skew-symmetric hilinear form (,). Let 
= 5po(V, [3) he the orthosymplectic Lie color algehra. 

(a) The image of a k-diagram d E T>]., under the map ^: Vj. Homg(T/®*^, V"^^) from 
(2.12), is the homomorphism given explicitly hy 

(ai (g) • • • (g) afc)*d = ^ da^ibi<g) ■ ■ ■<g)b,,, 

ii,---,ifcG-B 

where da^h is the weight of the k-diagram d with top vertices laheled hy ai, . . . , Gk 
and hottom vertices laheled hy hi,...,hk. (We have chosen to write as an 
operator on the right so that (h) will hold.) 
(h) Let m = dimV(o) and n = dimV(i) where V(o) = XlceG /3(c c)=i ^^'^ ^(i) ~ 
XlceG fi(c c)=-i Tien the map ^ extends to a homomorphism 

^■.Bkin-m) ^ HomJ,(y®^y®*^) 

of algehras. 

Proof, (a) In this setting the map pr^. is given by 

pj.^. I , y®*^ (g) (y*)®*^ 

1 I — > Vi^®---® Vi^ (g) t;'" (g) • • • (g) v'^ , 

l<ii,...,ik<N 

which can be seen by induction using (2.5). One computes from the definition in (2.12). 
The following sequence gives a representative pictorial example, but the general case is done 
in exactly the same fashion. For brevity in the pictures we reduce the notation even further 
and write 13 for Vi^ and for the dual vector v'^. 

Let Gi = v„^ for i = 1,2,3,4 and let us compute the action of for the diagram 




on the element 01020304 = Vn-^^Vn^Vn^Vm G (we omit the tensor symbols (g as another 
space-saving device). Note that the image of the diagram d under the unfolding map u of 
(2.1) is the one-factor 

u{d) = rr^w^^r^ . 

Following the definition of as a composite map and using the fact that Rvy(v (g w) = 
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-(3{w, v)w (8) V, we have 




pr^gjid®'' • • • 

ai aj 83 84 I > ii 12 13 14 i" i' i» a, 83 a^ 

21,22,23,24 



id®f«g,(-_p-i)®f«g,id®f« 




E ll****** 
i 1 12 13 i 4 F i" F F F aj a^ a^ 



-P(F-'v'\F-\'-)i, i,FV Qi^FV Qi^ *a, *a, 'a, *a. 





E -/3(^^-'^''S^''=)ii i2FV I 



^ ¥14 i" ai aj 83 84 



^ ii i2FV i3 




2l,22,23,24,Jl,j4 



^ wt 

2l,22,il,23 



^1 ^2 % ^4 




*'2-it'22t'ilt'2-3- 



\ ii 12 ji 13 / 



In doing these calculations it is helpful to note that deg( Fvi) = deg(t;j) = deg(t;') ^ = 
deg(F-^v')-^. 

(b) It follows from the definition of the weights of labeled diagrams that 

{vi,ViJ'$e = Fi^^i^Y Fj~j2VjiVj2, and {vi^Vi^)'$, = -l3{vi^,ViJvi^Vi^, (2.17) 
ii,i2 

for the case k = 2 and the 2-diagrams 
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For arbitrary k, 

= id®('-i) Cx) Cx) id®(*^-'-i), and 

where e,- and Si are the generators of the Brauer algebra in Theorem 2.2. Given these 
observations it is sufficient to check that and satisfy the relations in the statement 
of Theorem 2.2, for i = 1. All of these relations are easily verified by direct calculation. For 
example, the relation e{ = (n — m)ei follows from the computation 

tiM iij2 
tiM iij2 
i^i/2 i2 ii 

= ^'il,i2 Z ^^I^^S^^Ta Z -/5(^i2> t'i2)'5i2j2 
i^l/2 i2 

= (n - m)Fi^^i^ ^ t'^it'^2^£;,£2 

In this argument the critical facts used are that Fj-^ j^ = —l3(vj^ , Vji)Fj^ j-^ and that Fj^ j-^ ^ 
forces deg(t;jj = deg(t;jj-i so that i3{vy,,v^^) = l3{vj^,VjJ-'^ = l3{vj^,VjJ = ±1. 

The relation = id®*^ follows from the fact that , «2)/3(*2, *i) = 1, since /3 is 
symmetric. The relation ^g^^sj = ^s^^gj = can be derived from the same type of 
easy calculations. One shows that 

and the remainder of the relations are checked with similar calculations. | 

spo{V, l3)-mvanants m 

We retain the setup and notation of Theorem 2.16. If / is a 2A;-one-factor, let /j denote 
the 2A;-one-factor / with its vertices labeled (left to right) by the sequence h= (hi, ... , &2fc) 
of basis elements hi,. . .,&2fc £ B. Assign a weight to each edge and each crossing of the 
labeled one-factor /j as follows: 

(a) Weight the edge (h, h') by F~l,. 

(b) Weight each crossing by —(3{li , £2) where £1 is the label of a vertex that is adjacent 
to one of the edges, and £2 is the label of a vertex that is adjacent to the other 
vertex. Of the four vertices adjacent to the two edges that cross; £1 and £2 should 
be chosen to be the last two vertices (in order) when counting off the vertices left 
to right. 

Given a one-factor /j which is labeled by & = (&i, . . . , &2fc) define the weight wt(/j) of /j to 
be the product over all the weights of the crossings and edges. The proof of the following 
proposition is exactly analogous to the proof of part (a) of Theorem 2.16. 
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Proposition 2.18. Retain the setup and notation of Theorem 2.16. The image of a 
2k-one-factor f £ 02k, under the map $ : 02k (y«)2fc^B f^.^^ (2.11), is the invariant $/ 
given explicitly by 

$/=$/(!)= Wt(4)&i(g)---(g)&2fc, 

where the sum is over all possible labelings h= (&i, &2, • • • , &2fc) of the one-factor f. 



3. Maximal vectors in tensor space 



Let f) denote the diagonal matrices in 5p (V, l3) as in Section 1. An 5p (V, l3) -module 
M is said to have a weight space decomposition relative to f) if M = ©X]|ue()» where 

= {v ^ M \ hv = iJ,(h)v for all h G f)}. A vector in M is a maximal vector of 
weight A if Q+v^ = (0) and hv^ = X(h)v^ for all /i G f). In particular, the module V is an 
irreducible 5po(V, [3)- module with a unique (up to scalar multiple) maximal vector = ti, 
which has weight ci . All the basis vectors in 5 = {ti, t*i, . . . ,tr,t*,ui,u*i, . . . ,Ug,u*, (ms_|_i)} 
are weight vectors with weights given by 

wi{ti) = Ci = -wt(t*), i = l,...,r, 

wt{uj) = 6j = -wt{u*), j = l,...,s, (3.1) 

wt(Ms_|_i) = 0, if n = 2s + L 

If vi <E) ■ ■ ■ <E) Vk is a simple tensor in T/®*^ with Vi G B for each i, then its weight is just the 
sum of the weights of its factors. 

In this section we construct maximal vectors of T/®*^ using the elements Si, ei coming 
from the Brauer algebra. The maximal vectors in T/®*^ are often a help in locating its 
irreducible summands. We begin with a few combinatorial preliminaries. 



Young symmetrizers, contractions, and spo{V, j3)-suhmodules ofV"^^ 

Suppose A is a partition of k. The standard tableaux T of shape A are obtained by 
filling in the frame of A with the elements of /C = {!,..., so that the entries increase 
across the rows from left to right and down the columns. Let ST(IC) denote the set of all 
standard tableaux as A ranges over all the partitions of k. We associate two subgroups in 
the symmetric group Sk to each standard tableau T G ST(IC). The first is the row group 
Rt consisting of all permutations in Sk which permute only the elements of /C within each 
row of T, while the second is the column group C't of T or group of permutations moving 
only the elements of /C in each column. Then the element 

ST=lj2 '''M^m E ^ = E s9n(^)H^CSk (3.2) 

\ViGi?T / \7eCT / viGi?T 

7GCt 

has the property that there is some h(X) G that only depends on the underlying frame 
of T such that 4 = h(\)sT (see [We, Chap. 4, §2]). If yr = (l/h(\))sT, then = yr 
so that j/T is idempotent. We refer to yx as the symmetnzer determined by T . For each 
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T G ST{K) the space CSkVT is a minimal left ideal of the group algebra CSk and CSk = 
®J2TeST(ic) '^^kUT, (see for example, [We, Chap. 4, §3.4]). 

Starting at the left end of the first row and moving from left to right we compare 
the entries of two standard tableaux Ti and T2 of shape A. If the first nonzero difference 
ii ~ i2 is positive for corresponding entries ji in Ti and j2 in T2, then we say Ti > T2. If 
all corresponding entries in the first row are equal, then we proceed to the second row, etc. 
Thus, Ti > T2 if ii > j2 for the first pair of corresponding entries ji, j2 which differ. With 
respect to this ordering, we have the following lemma. 

Lemma 3.3. (See for example, [We, Theorem 4. 3D].) Let Ti, T2 G ST{K) he two 
standard tableaux of the same shape. If Ti > T2, then yTiUT-^ = 0- 



A partition A = (Ai , . . . , A^, 0, . . .) h / is said to be of (r, s)-hook shape if A^+i < s. If £ 
is a subset of K of cardinality /, then 



%ST r^s{i^) {T G ST(C) I T has an (r, s)— hook shape} 



(3.4) 



is the set of all standard tableaux with entries in C of shape A as A ranges over all partitions 
of / that are (r, s)-hook shape. 

Example. Suppose r = 2 and s > 2, and let A = (P, 2^, 4) = (4, 2, 2, 1, 1) so that 



Then A h 10 is of (2,s)-hook shape since A2+1 = 2 < s. The standard tableau T G 
nSTr,s{jC), where £ = {1, 2, 4, 6, 7, 8, 9, 10, 11, 14}, depicted below 



T 



1 


6 


11 


14 


2 


8 




4 


9 




7 






10 







determines two subtableaux 



1 


6 


11 


14 


2 


8 





id T(2) 



4 


7 


10 


9 





of shapes (4, 2) and (3, 1) = (2, 1, 1)' respectively. 
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We have seen in Section 2 that the Brauer algebra Bk{n — m) acts on V"^^ , and its action 
commutes with that of spo(y, [3) since it acts by spo(y, /3)-module morphisms. In particular, 
the diagrams d having all vertical edges form a subalgebra of Bk{n — m) isomorphic to the 
group algebra of Sk , and the resulting action of CSk on T/®*^ is by graded place permutations. 
We let Cp q denote the transformation on T/®*^ determined by the diagram in B].{n — m.) with 
a horizontal edge connecting the pih and gth nodes on both the top and bottom, and with 
every other top node connected to the one directly below it. Thus, Cpp^i corresponds to 
the element Cp in Section 2. We refer to the transformation Cp q as a contraction mapping. 
Clearly Cp q = Cq p, and we may suppose in working with contraction mappings that p < q. 
If p = {pj^, . . . and q = {qi, . . . ,qj} are disjoint ordered subsets of /C = {I, . . . ,k}, we 
set 

Cp,q_ = Cp^^q^ . . .Cp^^q^ for J = 1 , . . . , [A;/2j , (3.5) 

and we assume C0_0 is the identity diagram. Since the image C -^Vui^ ® w of e (see 

(2.17)) on is a one-dimensional trivial spo(y, /3)-module, we can draw the following 

conclusion (in Proposition 3.6 (a) below) that explains why these mappings are termed 
contractions: 

Proposition 3.6. (a.) V^^Cp^q = y®(*^-2) foj. all p,q = I, . . .,k with p ^ q. 

(b) Let T/®*^ be the subspace of all vectors in T/®*^ which are annihilated by all con- 
tractions Cp q, Then T/®*^ is an spo{V, /3)-submodule and an S^-submodule of T/®*^. 

(c) Let V'^'^Cp^q be the subspace of all vectors in V^'^Cp^q which are annihilated by all 
contractions Cg t with s,t G (p U qY = IC — (pU q). Then V^'^Cp^q is a 5po(V, [3)- 
submodule (isomorphic to y®(*^~2j)^ j _ j^j _ ^jjich is invariant under the 
action of the symmetric group S'(pug)=- 

Proof. Since each Cp q commutes with the action of 5po(V,l3), it determines an 5po(V, l3)- 
module endomorphism on T/®*^, and T/®*^, which is the intersection of the kernels of the 
maps Cp q, is thus an 5po(V, /3)-submodule. Its invariance under the action of Sk comes from 
the identity 

(^'^Cp^qa = C„(p-f^„(q-f (3.7) 

which holds for all a ^ Sk and which can be verified in Bk(n — m) by just multiplying the 
corresponding diagrams . Part (c) follows by combining (a) and (b). | 

Construction and linear independence of maximal vectors 

\f p = \^pi, . . . , and q = {qi, . . . ,qj} are disjoint ordered subsets of K = {1, . . . , A;}, 
let {p, q) = {{pi, qi), . . . ,{pj, qj)}, and denote by p{j) the set of all such {p, q). Assume 

LIJ 

P=UKi)- (3.8) 

i=o 

Using the contractions Cp q and the symmetrizers j/t, we now construct maximal vectors of 

ytpjk 
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Theorem 3.9. Suppose X = (Ai, . . . , Xi, 0, . . .) is an (r, s)-hook shape partition of some 
integer I such that k — I = 2j and < j < [A;/2J — 1. Let X = Aifi + • • • + A^fr + 
X'^_^.l6l + • • • + X'^_^_g6s denote the weight determined by X. Assume (p,q) G p(j) and fix 
T G TiST r^s{{p U qY) of shape X, Let T*-^-* and T*-^-* he the corresponding subtableaux of 
shapes X'--^^ = (Ai, . . ., A^) and A^^) = (A^+i, . . • , A^)' = (X'^_^i, . . ■,X'^_^g) respectively. Then 
9 = WT^p^qCp qUT is a maximal vector in V^'^Cp gyx of weight X where w^^p^q = wi ■ ■ ■ is 
the simple tensor defined by 

tl if i ^ p 

tl if i e q 

tj if i E (p U qY and i is in jth row ofT^^^ 

Uj if i E (p U qY and i is in jth row ofT^'^\ 

If n = 2s and ^(A^^)) = s, then 9° = w!^ ^ ^Cp qi/T is a maximal vector of weight X° = 
A — 2X'^_^_g6s where p q ^® simple tensor obtained from w^^p^q by replacing the factors 



itfi 



Proof. Let w = WT^p^q. First we observe that wCp q is anniliilated by all contractions having 
both subscripts in (p U qY because {v,v') = whenever v,v' G {ti, . . . ,tr,ui, . . . ,Us, 

(ug+i)}. Thus, WCp q G V^'^Cp^q and 9 = wcp qyx G V^'^Cp qyT- We also observe that the 
weight of 9 is the same as the weight of w, which is A, since the weights of v and v* sum to 
zero. Now wsgn(il')'4'J = w for ip G Rt and j G C't if and only if ip E Rt(i) and j G Rt(^)- 
That assertion follows from the fact that j3{ti,ti) = 1 for all ti and I3{ui, Ui) = —1 for all Ui, 
and the symmetric property of the bicharacter. Therefore, when 9 is expressed as a linear 
combination of the basis tensors, the coefficient of w in 9 equals h{\)~^\Rrp{i)\-\Rrp{2)\. Thus, 
6* 0. To prove that 9 is a, maximal vector, it suffices to show that it is annihilated by each 
of root vectors Xj in (1.12). Since the image of a contraction is a trivial spoiV, /3)-module, 
the sum of the terms with Xj acting on a pair of contracted slots is zero. Thus, Xj annihilates 
WCp q or produces a sum of tensors which are obtained by applying Xj to a noncontracted 
factor. Each of those tensors has one factor in {pyjqY whose subscript has been lowered by 
one or a factor ui which has been changed to t^. For such tensors, each of them denoted by 
V, we argue that vy^ = 0. 

First consider the case that ti has been changed to for 1 < i < r or that ui 
has been changed to t^. Suppose tp G Rt ■ Then there exists (a h) G Ct which permutes 
two factors of vip which are the same tj such that vtp^a h) = —vtl) (see (2.17)). Then 
sgn{il>){vil>)'^ = Y.-yeCT sgn{ij){vij){ab)j = -E^eCy sgn{ij){vij)j. Thus, we have 
E7GCT sgn(il')(vil>Y/ = for each ip G Rt and vyT = 0. 

Now consider the case that Ui has been changed to Then there exists (a h) G Rt 

permuting two factors of v which are the same Ui such that v(a h) = v (see (2.17)). As 
a result, YljjeRr ^^Hi^Yi^ = Eviei?^ ^^"((a b))sgn{ij){v{a b))ij = -J^ijeRr^Snii^Yi^- 
Therefore, E^/jei?^ sgn{Y)vY = 0, and vyT = 0. Thus, for all such v we have vyT = 0, and 
consequently Xj(wCp qyT) = (xjWCp q)yT = 0. 

If n = 2s and £(A(2)) = s, then the same argument as above proves that 9° is a maximal 
vector of weight A — 2X'^_^_g6s since in this case Xj for all j 7^ r + s — l,r + s acts on w° 
exactly as it acted on w. The action of Xr+s-i is zero since all the factors Ug have been 
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transformed to u* , while the action of x^+s on w" produces a sum of tensors each of which 
has one u* factor changed to Ms-i. Consequently, 9° is a maximal vector as claimed. | 

Now consider V^^Cp^q where {p,q) E p{\k/2\). When k is even, V^^Cp^q is the trivial 
spo(y, /3)-module k = 1, and when k is odd, it is isomorphic to V. If k is odd, then T G 
%ST r,s{{p U qY) consists of a single box with a number in it so that j/t = id. When k is 
even, then {p U qY = 0, T = 0, and we set t/0 = id. Let w^^p^q = wi ■ ■ ■ he the simple 
tensor prescribed by Theorem 3.9 in these cases. Then WT^p^qCp qyx is a maximal vector of 
weight when k is even, and WT^p^qCp qyx is a maximal vector of weight ei when k is odd. 
Moreover, V^'^Cp^q is generated by WT^p^qCp qi/T . To summarize we have: 

Corollary 3.10. For {p,q) £ p{[k/2\) 

^ ^P.i=\T/ if k IS odd, 

and WT^p^qCp qi/T is a maximal vector of V^'' Cp q of weight when k is even and of weight 
fi when k is odd, where w^^p^q is as in Theorem 3.9. 

Now we show that the maximal vectors constructed above are linearly independent if 
the rank of the algebra spo(y, /3) is sufficiently large. Suppose r + s >k and s > 2. Assume 
A = (Ai, . . . , Xi, 0, . . .) is an (r, s)-hook shape partition of / for some < / < A; and let A^^) 
and A(2) be the associated subpartitions. We argue that in this case £(A(2)) < s. Indeed, if 
£(A(2)) = s, then since A^ > £(A(2)) = s, |A| = |A(i)| + |A(2)| > r.s + s > r + s > k. This 
contradicts the fact that A partitions / < k. Thus, when r + s > k , we need only work with 
the maximal vectors of the form WT^p^qCp qyx . 

Theorem 3.11. Suppose that r + s > k. Then 

{WT \{p,q)ep, T E-HSTrAiP^lY)] 

is a hnearly independent set of maximal vectors. 



Proof. Vectors corresponding to different numbers of contractions are linearly independent 
because those vectors have different weights. As the weight is determined by the shape of the 
tableau T, vectors corresponding to tableaux with different shapes are linearly independent. 
Thus, it suffices to consider the (p,q)'s in p(j) for some fixed value of j, and the tableau T 
of a fixed shape A. Suppose 

aT,p,qWT,p,qCp^q_yT = 0, 

(£,i)Gp(i) TensTr.siipyqy) 

where the sum is taken over all the tableaux T G TiST r,s{{p U qY) of some given shape 
A = (Ai, . . . . . .), which is an (r, s)-hook shape partition. We suppose that A^"'^) = 

(Ai, . . . , Ar) and A^^) = (A^+i, . . . , A^)' = [X'^j^i, . . . , X'r+g) the partitions determined by 
A, and let li = and £2 = Among the simple tensors involved in the expression 

for a fixed WT^p^qCp q there is a unique tensor CT,p,q matching the following description: 
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First when I2 = 0, the sequence of positions pi, qi, p2, q2, ■ ■ ■ , Pj , Qj in CT,p,q con- 
tains the first 2j terms of the sequence t£i+i,t£^_|_]^,t£i+2,^£j_|_2' • • • ; ^r, fi, ^1, • • • , ^s, fj- 
Note there are sufficiently many vectors to make this work since r + s — j > k — j > 
k — 2j = |A| > £1 implies r + s — £1 > j. Similarly when £2^0, then the sequence 
of positions pi,qi,p2,q2, ■ ■ ■ ,Pj ,qj in CT,p,q contains the first 2j terms of the sequence 
M^j+i ) ^£2+1 ' "^2+2 ) ^£2+2 ) • • • ) , Ms , and there are enough vectors in this case since r-\-s—j > 
k - j > k - 2j = \X\ > £1 + £2 = r + £2 so that s - £2 > j. 

Then it follows that 

X] X] aT,p,qCT,p,q}jT = 0, 

il,q)ep(j)Te-HSTr.siipyq)'') 

where the sum is over T of shape A. The simple tensors in the above corresponding to 
different choices of (p,q) are linearly independent, for the positions of the factors 
and , or Ui(x('-'))+i and u*|■^^2)'f_^.i are sufficient to distinguish them. Thus for any 

{p,q) e p(j), 

^ aT,p,qCT,p,q}jT =0. 

TensTr,sapuq_r) 

If some coefficient in (3.12) is nonzero, then since the sum is over tableaux of the same shape, 
we may choose T' minimal in the order with the property that cit' ,p,q ^ 0. By Lemma 3.3, 
T > T' implies that UtUt' = 0. Thus applying j/y/ to (3.12), we obtain 

= ttT' ^p^qCr' ,p,q_yT'yT' = aT' ,p_,qCT' ,p_,q_yT' ■ 

Since the vector CT',p,q appears in CT',p,qyT' with coefficient equal to h(\)~^\Rrpi(i) \ ■ |i?y/(2)| 
as we discussed in the proof of Theorem 3.9, Ct' ,p,qyT' 7^ 0. Hence, no such T' exists, and 
ttT^p^q = for all T and all (p, q). | 



4. The spo(y, /3)-modules and their characters 



Defimiion of the modules T 

Let 5p (V, l3) be the orthosymplectic Lie color algebra and assume that m = dimV(o) 
and n = dimV(i). Recall that by Theorem 2.16 there is an action (which we write on 
the right) of the Brauer algebra B].(n — m) on T/®*^ which commutes with the action of 
spo{V, l3) on y®*^. For convenience we write wd rather than w for the action of the 
element c? G B].{n — m) on w G y®*^. 

Suppose that |n — m| > k. Then by the work of Wenzl [W] we know that the Brauer 
algebra Bk(n — m) is semisimple and has simple summands indexed by the partitions in the 

set 

Bk = k -2h \ h = 0,l,..., [k/2\}. 
More specifically, there are positive integers d\, X ^ Bk , and an isomorphism 

T:Bkin-m) Md^in), 

AGS;, 
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where Md^(K) denotes the full matrix algebra of d\ x d\ matrices with entries in k. 

For each \ E Bj. and \ < P,Q < d\, \ei Epq denote the matrix unit in the Ath 
block of Md^in) which has a 1 as the (P,Q) entry and zeros everywhere else. Let 

= T-\E^q) and define 

T^' = V^'^e^pQ, (4.1) 

where A' denotes the partition conjugate to A, and the action of Bk{n — m) on T/®*^ is as 
given in Theorem 2.16. Then T'^ is a g-module since the action of g on T/®*^ commutes with 
the action of ep g G B].(n — m); however, the module T'^ may be (0). 

Proposition 4.2. If |n — m| > k, then the following hold: 

(a) The Q-module T'^ is well-defined (up to Q-module isomorphism it does not depend on 
the choice of P and Q). 

(b) As modules for q x Bk(n — m), 

where B\ is the irreducible Bk(n — m)-module labeled by A for each A G 5^. 



Proof. We shall only sketch the proof of this standard result, 
(a) The map 



„A . T/(g)fc A T/(g)fc A 



is a 0-module isomorphism with inverse given by 



„A . T/(g)fc A T/(g)fc A 

^'^R,S ^ ^'^R,S'^S,Q = (""5i?,p)'5p,Q- 

(b) For each A G 5^, the irreducible B].{n — m)-module B\ has a basis {eg | 1 < Q < 
dx}, and the action of Bk{n — m) on this space is given by 

^Q^R,S = '^A.M'^Q.Pei 
Note that since 1 = X^Ags X]i<Q<dj '^Q Q may write 

For convenience let us fix a particular index, say _P = 1, for each A G 5^ and set T'^ = 
y(g)fcgA Now one can check that the maps 

$: 0^T^'c3 5, ^ 1^®' = Ea Eq 
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"' = EaEq"'4,q ^ Ea Eq("' 4,1)^1,1 ® 4 

are g x Bk(n — m)-module homomorphisms which are inverses of each other. | 



Characters of the Brauer algebras 

If di is an mi-diagram and 3,2 is an m2-diagram, then di ®d2 is the (mi + m2)-diagram 
obtained by placing di to the right of d2. Let e denote the 2-diagram 



e = 




(4.3) 



and let 7™ denote the m-diagram 



7m = 




For a partition = {ni, ^2, ■ ■ ■ , l^t), let 7^ = (g) ® ' ' ' ® 7mi- We have the following 
results from [R]. 

Theorem 4.4. 

(aj Ifrj^O, then any character of the Brauer algebra Bk(ri) is completely determined by 
its values on the elements e®-* Cg) 7^, for j = 0,1, . . . , [A;/2J and jj, \- k — 2j . 

(b) Assume that k C C so that ry G C. If rj ^ Z or if rj ^ Z and |ry| > k, then the irreducible 
characters of Bk(ri), 

which are indexed by the partitions X in the set Bk = {X \~ k — 2h \ h = 0, I, . . . , [A;/2J }, 
are given by 

Xi(,)(e«^' ® 7.) = E f E ^A,.) Xk-Jp)^ 

i^DX \0 even / 

where x's^_2-i ^1^^ denotes the irreducible character of the symmetric group Sk-2j labeled 
by v \- k — 2j evaluated at the conjugacy class indexed by jj, \- k — 2j, and cj( g is 
the Littlewood-Richardson coefEcient (see (4.9) below), and the inner sum is over all 
partitions 6 with even parts. 



Weighted traces 

Assume that the basis B = Bq U Bi of V is as in (1.8). Let {zjjjgs be a set of 
commuting variables indexed by the elements of B such that 

zj. = Zj"\ for all & e 5, (4.5) 
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(in particular, Zu^_^-^ = 1). Define an operator D on T/®*^ by 

D(hi (g) h'j (E> ■ ■ ■ (E> hk) = Zi^zi,^ • • • zj^ (&i (8) &2 ® • • • ® 

for hijh'z, ■ ■ -hk G B, or simply D(hi---hk) = zi,-^zi,^ ■ ■ ■ zi,^(hi ■ ■ ■ hk) deleting the tensor 
symbols. Since (by a brute force check) D commutes with the action of the generators e,- 
and Sj- , 1 < i < A; — 1 of the Brauer algebra, it follows that D commutes with the action of the 
Brauer algebra B].(n — m) on T/®*^. Using D we define the weighted trace of d E B].{n — m) 

by 

wtr(rf) = Try®.(rf_D). (4.6) 



Lemma 4.7. 

(a) If d is a k- diagram, then 

wti(d) = ^ Zi^zi,^ ■ ■ ■ Zi^^di^i, 

where h = (&i, &2, • • • , &fc) and rfj j denotes the weight of the diagram d labeled on the 
top by hi, . . . ,hk G 5 and on the bottom also by hi, . . . ,hk. 

(b) The map wtr is a trace on Bk(n — m), i.e., wti(did2) = wti(d2di), for all di,d2 G 
Bk(n - m). 

(c) If di is a k-diagram and d2 is an l-diagram then wii{di ® d2) = wtr((ii)wtr((i2). 

(d) If d E Bk(n - m) then 

wtr(rf) = ^ xl(„-™)(rf)char(T^), 

AGS;, 

where XB,,(n-m) denotes the irreducible character of Bk{n — m) indexed by A' G 5^ 
and char(T^) = TrTA(_D). 



Proof, (a) This is a direct consequence of the definition of the weighted trace. 

(b) Since the actions of D and B].(n — m) commute on T/®*^, it follows that 

wtr(rfirf2) = Tr(rfirf2-D) = Tr(rf2-Drfi) = Tr(rf2rfi-D) = wtr(rf2rfi). 

(c) This comes from (a) and the fact that if &i, . . . , 6^, h^+i, ■ ■ ■ , h^+i G B, then 

D{hi ■ ■ ■ hkhk+i ■■■h:)= [D{hi ■■■hk)) (D{hk+i ■ ■ ■ hk+t)) 



and 



[hi--- hkhk+i ■ ■ ■ hk+i^ (di (g) d2) = ((hi ■ ■ ■ hu)d^ {{ik+i ■ ■ ■ hu+i)d2^ ■ 



(d) It is clear that T'^ is also invariant under the action of D as the actions of D and the 
Brauer algebra Bk(n — m) commute. Thus, all the statements of Proposition 4.2(b) hold 
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with D in place of g. By taking the trace of the operator dD on each side of the isomorphism 
in Proposition 4.2(b) (with g replaced by D) we have that 

wtr(rf) = 1iy.,.{dD) = J2 TrB„(rf)TrT.(i?) = ^ xl(„-™)(rf)char(T^). | 

In view of Lemma 4.7(c) and Theorem 4.4(b), it is sufficient to compute the weighted 
traces wti(d) when c? is a fc-diagram of the form e®-* Cg) 7^. This is done in the following 
lemma. 

Lemma 4.8. Let jj, = (jii, 1^2, ■ ■ ■ , /J-t) be a partition ofk — 2j for some j G {0,1,..., [A;/2J }. 
Let e®-* (8) 7^ be the k-diagram coming from (4.3). For each positive integer £ suppose 

beBi beBo 

Then 

wtr(e®-'' (g) 7^) = (n - m.yp^{Z). 



Proof. From the definition of the labeled diagrams and the properties of the matrices Fb 
and Fg^ in (1.9a,b) we see that 

r-l iibl = b2eBo, 
= KX^iui2 = j 1 if &t = &2 e 5i , 

I otherwise, 

where h= (hi, 62)- If & = (&i , &2, • • • , &fc) then 

(7.)m = n ( - p(hA)) = I 'y-' = • • • ' 

10 otherwise. 

z = l ^ 

Consequently, we have 

wtr(e) = ^ zj^zj^ej^j = ^ (-l)zjzj. + ^ zjzj. = ^ (-1) + ^ 1 = n - m, 

wtr(7.) = Y: + E 

beBo beBi 

The result now follows since by Lemma 4.7(c) we have that 

wtr(e®-' (8) 7^) = wtr(e)-'wtr(7^Jwtr(7^J • • •wtr(7^J. | 

Symmetric functions and a combinatorial description o/char(T'^) 

We adopt the notation for partitions and symmetric functions found in [M]. Let us 
begin by recalling a few definitions. 
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Suppose Y = {j/i, . . . , j/g} is a set of commuting variables ordered by j/i < t/2 < • • • < J/g, 
and assume A and /j, are partitions such that C A. A column-stnct tableau of shape A/// is 
a filling of the boxes in the Ferrers diagram of A/// with j/j's such that the j/j 's are weakly 
increasing (left to right) across the rows and are strictly increasing down the columns. 
Associated to a column-strict tableau T of shape A/// is the monomial ]p- , which is the 
product over all boxes of A/// of the elements yj in the boxes. The skew Schur function is 
the sum 

T 

over all column-strict tableaux T of shape A///. The (ordinary) Schur function sx{Y) is just 
the skew Schur function s\i^{Y) in the special case that jj, = ^. The Ltttlewood- Richards on 



coerncie 



nt is the nonnegative integer ^ defined by 



s,{Y)s,{Y) = Y,ci.s^{y)- (4.9) 



The Littlewood-Richardson coefficients exhibit certain symmetries such as 

cl,. = 4y (4-10) 

It is a standard fact that the skew Schur functions can be written in terms of the ordinary 
Schur functions via the relation 

■^xi,{Y) = Y.cl,s,{Y). (4.11) 



A partition p can be specified by its Frobenius notation p = {qi, . . . ,qp\ri, . . . , Vp) where 
the main diagonal of p consists of p boxes ^ < i < P, and there are qi boxes to the 

right of (i, i) in the ith row and ri boxes below (i, i) in the ith column. When p partitions k, 
it is customary to write \p\ = k. We have the following identities involving Schur functions 
(see [M, pp. 77-79]): 

n (1 = E(-1)'''^%(^)' (4-12) 

l<i<j<q p 

where the sum is over all partitions of the form p = (ri -|- 1, . . . , -|- 1 | ri, . . . , r^), ri < q — l, 
in Frobenius notation; 

n (1 =E(-i)'''^'^-(^)' (4-13) 

l<i<j<q TT 

summed over the partitions tt = (ri — 1, . . . , — 1 | ri, . . . , r^), ri < g — 1, in Frobenius 
notation; and 

n {^-ymr'= E '^(Y), (4.14) 

IKiKj <.q even 

where the sum is over all partitions 6 with even length rows (all parts even). By using the 
above identities one can easily prove the following 
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Proposition 4.15. 



\ /9 / \9 even 



^A,«y-^.,A, (4.16) 
wiere tie sums over p and 9 are as in (4.12) and (4.14), respectively. 

Proof. By using (4.12), (4.14), and (4.9) to expand, we get 

1 < i<.j <.q 1 < i<.j < q 



E^^(nE(E<.) fE(-i)"'^^< 

r \e even / \ p 



and by comparing coefficients of the Schur functions on each sides, we get identity (4.16). 
I 

Let us recall the definition of the hook Schur functions given in [BR] and [Se], which 
describe the characters of the irreducible gl(T/, /3)-modules appearing in T/®*^ when G = Z2, 
and /3(a,b) = (—1)"'; that is, for the general linear superalgebra. Order the variables zj, 
6 e 5 = 5o U 5i , by 

Zti < Zt* < ■ ■ ■ < Zi^ < Z(» < Ztjj < Zt,» < • • • < Zt,^ < Zt,» < (Zt,^_|_j). 

A httahleau of shape A is a filling of the Ferrers diagram of A with elements of B such that 

(1) the portion of the diagram filled with Zj's is the diagram of a partition C A, 

(2) the Z('s are weakly increasing (left to right) along rows and strictly increasing down 
columns. 

(3) the weakly increasing down the columns and strictly increasing left to right 
across the rows. 

If T is a bitableau of shape A, then Z"^ is the product over all boxes of A of the elements zj 
in the boxes, and the hook Schur function (see [M, Chap. I §3 Ex. 23-24 and Chap. I §5 
Ex. 23]), is given by 

^A(i) = E-^ 

T 

where the sum is over all bitableaux of shape A. 

The hook Schur functions satisfy the following identities (see [M, Chap. I (4.3), (4.3'), 
(7.7) and §3 Ex. 23]): 

E.^^ riiGBi 111 = 1(1 + -^iW) I-tX 
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where s\(Y) denotes the ordinary Schur function in the variables yi, . . . ,yq. li \ \- k, then 



sxiZ) = J2^^P,iZ), (4-19) 



3/11 



where (i) Xs (a*) denotes the irreducible character of the symmetric group indexed by the 
partition X \- k evaluated at the conjugacy class labeled by the partition jj, \- k, (ii) 3^ is 
the cardinality of the centralizer of an element of cycle type jj, in Sk, and (iii) Pfi(Z) is the 
function which is defined by 

k(Z)= ^ zf- ^(-1/zf, and p,{Z)=p,,{Z)---p,XZ). 

beBo beBi 

The functions Pij_{Z) defined in Lemma 4.8 satisfy Pi{Z) = (—lY(—pi(Z)) for all £ > 0, 

and p^{Z) = Uii-'^T'i-Pf^.iZ)) = i-lf-^^'^'>p^iZ), for a partition ^ \- k. Then (4.19) 

and the fact that Xs,(/«) = (-l)*'"^^''^Xsl (/«), (^^^ fo^" example, [M, Chap. 1, §7, Ex. 2]) 
gives 

■sx(Z) = J2^^^p^(Z). (4.20) 

Let Zq = {zj I h G Bo} be ordered as in the definition of the hook Schur function. A 
symplecttc tableau of shape A is a filling of the boxes in the Ferrers diagram of A with zj's, 
h ^ Bo, such that 

(1) the zj's are weakly increasing (left to right) along rows and strictly increasing down 
columns, 

(2) the elements Zj^ and Zf never appear in a row with number greater than i. 

Associated to a symplectic tableau T of shape A is the product z'^ of all the elements zj in 
the boxes of A. The symplectic Schur function is defined by 

spx(Zo) = Y,^^' (4-21) 

T 

where the sum is over all symplectic tableaux T of shape A. 

Assume q = Card(Y) is sufficiently large, i.e. q >> r, and define functions sc\(Zo) by 
the identity 

Y,sc,{Zo)s,{Y) = - / n i^-y^Vj)- (4-22) 

When A is a partition such that £(X) < r, then sc\(Zo) = sp\(Zo) and these polynomials are 
the characters of the symplectic group Sp(2r) and its Lie algebra sp(2r). The combinatorics 
of these functions is discussed in [Sul]. 

Analogously suppose Zi = {zj | h G Bi} and assume q >> s. Define functions sh\(Zi) 
by the identity 

J2.sh,(Z,)s,(Y) = - / n (4-23) 
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When Card(5i) is odd and A is a partition such that £(X) < s, then the polynomials sh\(Zi) 
describe the characters of the orthogonal group S0(2s + 1) and its Lie algebra so(2s + 1). 
In this special case these functions have a combinatorial description similar to that of the 
symplectic Schur functions (see for example, [Su2]). 

Theorem 4.24. Let Z be the set of variables {zj} indexed by the elements h ^ B = BqUBi 
and letY = yi, . . . be an auxiliary set of variables such that q >> r, s. Then the following 
identities give equivalent definitions of polynomials sc\(Z): 



(a) sc,{z) = y: (j:i-ir^'c'J 

iiCA \ Jr / 



where (Z) denotes the hook Schur function of shape fi, and ^ is the Littlewood- 
Richardson coefEcient. The inner sum is over all partitions of the form tt = 
(ri — 1, . . . , Tp — 1 I ri, . . . , Tp), ri < q — I, in Frobenius notation. 



rCA' \ p I 



(b) sc,iZ)= >^ 

where the inner sum is over all partitions of the form p = (ri + 
1 I ri, . . . , Vp) , ri < q — 1, in Frobenius notation. 

(c) }_^scx(z)sx(Y) = - — 11 (1 - y^y^)- 

(d) }_^scx(z)sy(Y) = - — 11 (1 - y^y^)- 

(e) .scx(Z) = ^ det {hx,-i-j+2(Z) + hx,-i+j(Z)). 

where hi(Z) = S(^fj(Z), and (£) is the partition of £ with just one part. 

(f) scx(Z) = \ det [{tx'-i-i^2{Z) - tx'-i-j{Z)) + {tx'-i^i{Z) - eA'_-i+j-2(^))) , 
where e.t{Z) = S(^-yv^{Z), and (1^) is the partition of £ with all parts equal to 1. 

(g) scx{Z) = det {ex'^-i+j{Z) - ex'^-i-j{Z)) . 

(h) scx{Z) = ^ sc^{Zo)sx'/^,'{Zi), 

where sxij^'iZi) is the skew Schur function in the variables zi,, & G Bi. 

(i) scx{Z) = ^ sb^-{Zi)sx/^,{Zo), 

where sx/i_i(Zo) is the skew Schur function in the variables zi,, h G Bq. 
Proof. Let us begin by proving (a) <S=^ (b), which follows from the calculation, 



sex 



/ \ tt' / 

= E fE(-i)'"''^^>) ^.(^) = E fE(-i)'"''<.) ^r-{z). 

\ P / T \ p / 
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Next we prove (c) using (a): 



A \,fi \ IT 

/II A,7r 

\ ^^ J l<i<j<q 



by (4.9), 
by (4.13), 
by (4.17). 



The proof of (d) using (b) is virtually the same using (4.11) and (4.16). 

Now let us argue that (c) implies (e): For each partition A = (Ai, . . . , A^), Ai > A2 > 



> Ag > define 



where the matrix (yf''^^ ) is q x q. Then as(Y) = Yii<i<j<qiyi ~ %)i ^^'^ Schur 
function s\(Y) can be written as 



ax+6(Y) 
as{Y) ' 



([M, Chap. I, §3.]). Let us write j/'^"'"'' = J/i '"'"^""'^ 1/2^"'"*"^ ■■■yq''. If /(Y) is any symmetric 
function in the variables j/i, . . . , j/^, then the coefficient of s\(Y) in the expansion of f(Y) 
in terms of Schur functions is given by 

[f(Y)UY) = [f(Y)as(Y)]y.^. . 

Thus, 



scx(Z) 



J2scxiZ)s4Y) 
J2scx(Z).sx(Y)as(Y) 

A 

YibeBo ni=i(i--2i%-) 



11(1 - yiyj)Y[(yi - yj) 
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where 



i<j 
i<j 



det (y, '+y, 

idet (yr^-'+yr 



-(i-i) 



Now from (4.17) we see that 



ihdBoy^ ^>>y') A r>0 r>0 



Thus, we obtain 



scx(Z) 



ldet(y^+^-' + yr)l[\Y,hAm 

i=l \r>0 



idet 



r>0 



= \ det \ hx,-i-j+2{Z) + hx,-i+j{Z) 

Identity (f) is gotten from (d) by a similar argument. 

To show (f) implies (g), add columns j — 2, j — 4, j — 6, ... to column j for each j. That 
is, add column 1 to column 3, add column 2 to column 4, add columns 1 and 3 to column 
5, add columns 2 and 4 to column 6 and so on. The result follows immediately. 

To derive (h) from (c), just compare the coefficients of s\(Y) on both sides of the 
following equation: 



J2scx(Z).sx(Y) 



11(1 - yiyj) 



YlieBo 111=1(1 - -^iW) 
(^■sc,(Zo).s,(Y)^ (^■s,,(Z,).sAY)^ by (4.22) 

J2sxiY)J2sc,iZo)J24,.s,'iZ,) by (4.9) 

A /II f 

^SA(y)^sc,(Zo)^c;;;_,,s,-(^i) by (4.10) 

A /II f 

Y,sx(Y)Y,sc,(Zo)sy/^.(Zi) by (4.11). 
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Obtaining (i) from (d) just amounts to a similar argument comparing the coefficients 
of s\(Y) on both sides of the following equation: 

Theorem 4.25. Assume \n - m\ > k, and for X \- k - 2h, h = 0, 1, . . . [A;/2J, let be 
the 5po(V, l3)-module in (4.1). Let sc\(Z) be the function defined in Theorem 4.24 and let 
char(T'^) be as defined in Lemma 4.7 (d). Then 

char(T^) = scx{Z). 

Proof. Suppose /j, \- k — 2j for some j = 0, 1, . . . [A;/2J . Then by Lemma 4.7 (d) and Lemma 
4.8, 

(n - myp^iZ) = J2 Xl(„-™)(e®^' ® 7M)char(T^). 

AGS;, 

Thus, it follows from Theorem 4.4 that 

MZ)= E E f E ^Iv)x5_,(/^)char(T^). 

AG-Bfe even / 

By (4.20) and the orthogonality of characters for the symmetric group we have, for ly h k — 2j, 
^^(^)= E E E(E^a-.1 "^--^"]"^--^"^ har(T^) 

fi\-k-2j ^^g^wDX' \e even I 

= E E ( E ^A'.'' ) char(T^)^., . = E ( E char(T^). 

AGSi'^^^' even / even / 

Thus, by Theorem 4.24 (b) and (4.16) it follows that 

-^(^)=E fE(-i)'''^'<.) ^^'(^) 

= E E fE(-l)'^'^'<:.) ( E <=har(T^) 

AG-Bfc ^ P / \S even / 

= (5^/_A'char(T'^), 
for all partitions X\- k — 2h, h = 0, I, . . . , [A;/2J . | 

Remark 4-S6. It follows immediately from the definition of the skew Schur functions, the 
definition of sp\(Zo) in (4.21), and Theorem 4.24(h) that the functions sc\(Z) for certain 
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partitions A can be expressed as a sum over weights of tableaux which have a symplectic 
part and a row-strict part. It is this interpretation which allows us to develop an insertion 
scheme modelling these functions in the next section. 



Up-down tableaux and s'^po -tableaux 

Fix positive integers r and n. An up-down tableau of length k and shape A is a sequence 
of partitions A = (A*^, A^, . . . , A*^) such that A*^ = and A*^ = A and, for each i = 1, . . . ,k, 
y is obtained from A'~^ by either adding or removing a box. An up-down (r, n)-tableau is 
an up-down tableau A = (0 = A*^, A^, . . . , A*^ = of length k and shape A such that each 
A' is an (r,n)-hook shape partition, that is, < n. If A*^ = A is the final shape, then 

\\- k-2h for some <h< [k/2\ . 

Example. When k = 7 , 



is an up-down (r, n)-tableau of shape A = (1, 0, . . .) for all r > 1 and n > 1. 

Let Bo = {ti,tl, . . . ,tr,t*} and Bi = {vi, . . . ,v„} and let B = BqUBi. Here we do not 
need to distinguish between n odd or n even and for that reason we do not use the notation 
ui,ul, . . . ,Us,u*, (ug+i) for the elements of Bi as we have done in previous sections. Order 
B as follows: 



Suppose m = 2r and n are fixed. An 5po(m, n)-tableau T of shape A (or simply an 5pO- 
tableau for short) is a filling of the boxes in the Ferrers diagram of A with entries from B 
such that 



(spo.l) the subtableau 5 of T obtained by taking all the boxes with entries from 
Bo is a column-strict tableau of partition shape (its entries are weakly increasing 
from left to right across each row and are strictly increasing from top to bottom 
down each column), and the entries in row i are > ti for each row in S; 

(spo.2) the skew tableau T/S is row-strict (its entries are strictly increasing from 
left to right across each row and are weakly increasing from top to bottom down 
each column). 



In an spo(m, n)-tableau the entries in row i for i > r-|- 1 necessarily belong to Bi, and since 
the skew tableau T/S is row-strict, the underlying partition must be an (r, n)-hook shape 
partition. 



5. Tableaux and an insertion scheme 




B = {ti < tl < t2 < t*2 < . . . < tr < t* < Vi < . . . < Vn}. 
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Example. Let 



T 



h 


f* 


i2 


V3 


V4 


f* 


f* 


Vl 


V3 




t3 


t3 








ti 











Then T is an spo(m, n)-tableau for all m > 8 and n > 4 where 



S 



tl 


f* 

n 


t2 


f* 

^2 


'^2 




t3 


t3 




ti 







and T/S ■ 









V3 


V4 






Vl 


V3 

















Theorem 5.1. If T is an 5po(m, n)-tableau, replace each entry by the corresponding 
variable zj as defined in (4,5) and let z'^ be the product of the entries in the result. Let 
sc\(Z) be the polynomial defined in Theorem 4,24, Then 

T 

where the sum is over all the spo(m, n)-tableaux of shape A. 



Proof, Suppose T is an spo(m, n)-tableau, and let S denote the subtableau of T containing 
the entries ti,t\, , , , ,tr,t*. Necessarily the number of rows in 5 is < r. When the entries 
ti,tl, i = 1, , , ,,r oi S are replaced by the elements Zt^, Zt* , then the result is a symplectic 
tableau as defined just before equation (4.21). Likewise, if we substitute the element z^^ for 
the entry Vj, j = I,, , ,,n, in T/S and transpose the resulting tableau, what is produced 
is a column-strict tableau. The result then follows from the definition of the skew Schur 
function, (4.21), and Theorem 4.24(h). | 

Assume 

Wk is the set of words of length k in the alphabet B, and 

Vk is the set of pairs (T,A) consisting of a spo(m, n)-tableau T of shape A and an 
up-down (r, n)-tableau A = (A*^, A^, . . . , A*^) of length k and shape A. 

We shall prove in Theorem 5.5 that there is a bijection between Wk and Vk- Accomplishing 
this requires introducing a few more definitions. 

Punctured tableaux and the maps jeu and injeu 

A punctured tableau is a tableau with exactly one empty box. A punctured s'^po -tableau 
is a punctured tableau obtained from an spo-tableau by removing the entry from exactly 
one box. If a partition A contains a box at position and there is no box at locations 
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(i, j + and j) in A, then is said to be a corner oi \. When a punctured tableau 

has its empty box at a corner, we may identify the punctured tableau with the tableau 
obtained by removing the empty box. 

In Lemma 5.2 below we shall define operations se and nw on the set of punctured 
spo-tableaux, but first we make the following definitions. 

(i) S£ is the set of punctured spo-tableaux whose empty box occurs at a corner. 

(ii) MW is the set of punctured spo-tableaux whose empty box is in the first column, 
and either the empty box is in the first row or the tableau obtained by switching 
the empty box with its neighbor to the north (the one immediately above it) is not 
a punctured spo-tableau. 

Lemma 5.2. (Compare [Be, Lemma 2].) Let T be a punctured 5pO-tableau. 

(i) Suppose T ^ S£. Then there is a unique way of switching the empty box of T 
either with its neighbor to the east (the box on its right) or with its neighbor to 
the south (the box immediately below) so that the resulting tableau is a punctured 
5pO-tableau. We denote the resulting tableau by se(T). 

(ii) Suppose T ^ AfW. Then there is a unique way of switching the empty box of T 
either with its neighbor to the north or with its west neighbor (the box on its left) 
so that the resulting tableau is a punctured 5pO-tableau. We denote this resulting 
tableau by nw(T). 

(Hi) IfT ^ S£, then nw{se{T)) = T, and ifT ^ MW, then se{nw{T)) = T. 

Proof. Assume the neighbors of the empty box are as pictured below, where one or more of 
the boxes with letters may be absent. (We are not necessarily displaying the entire tableau 
rather just the neighbors of the empty box since they are all that matter in the argument.) 



T: 



X 


a 


y 


h 




c 


z 


d 


w 



First consider the case that all the neighbors of the empty box are present. We note that 
a < c since a < y < c, and at least one of the inequalities is strict. Similarly h < d holds 
since h < z < d and at least one of the inequalities is strict. 

To prove (i) consider the tableaux 



e(T) 



X 


a 


y 




X 


a 


y 


h 


c 




and s(T) = 


h 


d 


c 


z 


d 


w 




z 




w 



If c = c? G 5o, then only s(T) can be an spo-tableau. If d is in row i of T, then since T is 
an spo-tableau, d > ti > ti-i so that condition (spo.l) is fulfilled in s{T). If c = c? G 5i, 
then only e(T) is an spo-tableau. If c > rf, then only s{T) is an spo-tableau, while if c < rf, 
then just e(T) is an spo-tableau. The tableau se{T) is s{T) or e(T), whichever one is an 
spo-tableau. (Note se{T) should not be confused with s(e(T)).) 
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For (ii) consider the tableaux 



w(T) 



X 


a 


y 




X 




y 




h 


c 


and n(T) = 


h 


a 


c 


z 


d 


w 




z 


d 


w 



If 6 = a G 5o and h is in row i, then ti < h = a < d and only n(T) is an spo-tableau. If 
6 = a G 5i, then h = a < c, and is an spo-tableau but n(T) is not. When a > h, then 

just n(T) is an spo-tableau, and when a < h, then only is an spo-tableau. The tableau 

nw(T) is either n(T) or w{T), whichever happens to be an spo-tableau. 

To prove (iii) suppose that T ^ S£ and se{T) = s{T). By (ii) nw{s{T)) is either 
n{s{T)) or w{s{T)) depending on which is an spo-tableau, and clearly n{s{T)) = T is an 
spo-tableau. Thus, nw{se{T)) = n{s{T)) = T in this case. Similarly, if se(T) = e(T), then 
nw{s{T)) = w{e{T)) = T . For T ^ MW , using (i) we see that se{nw{T)) = T . 

Finally, we note that when the empty box of T has only one of its neighbors to the east 
or south, arguments similar to ones above can be used to show that se{T) is obtained by 
switching the empty box with the available neighbor. Also, when the empty box of T has 
just one of its northern or western neighbors nw(T) is obtained by switching the empty box 
with the available neighbor. In these cases, (iii) holds similarly. | 



Let S£ and AfW be the sets of punctured spo-tableaux described just before Lemma 
5.2, and define maps jeu:AfW S£ and injeu: S£ MW as follows: 

(i) For T G MW , there is a least integer j > 1 such that applying se to T j times 
gives a tableau in S£. Then jeu(T) (se)-' (T). 

(ii) For T G S£, there is a least integer i > I such that (nw)'(T) G AfW. Then 
injeu(T) (nw)'(T). 

The following is an immediate consequence of Lemma 5.2. 



Lemma 5.3. The map jeu is a bijection from AfW to S£ with inverse mapping mjeu. 

Example. The sequence below illustrates applying the mapping jeu to a tableau in AfW 
and ending with a tableau in S£. 



f* 


i2 


i2 




f* 


i2 


i2 




f* 


i2 


i2 




f* 
l^l 


i2 


i2 










t3 




V2 




t3 


t3 


V2 




t3 


t3 


V2 








se 








se 








se 








t3 


Vl 


V2 






Vl 


V2 




Vl 




V2 




Vl 


V2 




V2 








V2 








V2 








V2 





If we take the rightmost tableau in this sequence as the initial tableau and perform nw 
successively, then the sequence moving from right to left corresponds to applying injeu. 



Insertion of a letter mto an s'^po -tableau 
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Let T be an spo-tableau, and assume a ^ B. We define an algorithm consisting of a 
sequence of steps which inserts a into T to yield a tableau (a T). 

(1) Start with h = a and i = j = 1. 

(2) If & G Bo, then insert h into the ith row of T as follows: If there is an entry in 
row i which is greater than h, then displace the leftmost such entry and insert h 
into its box except in the following case. If & = ti and there is an t* in the ith 
row, then replace the leftmost t* in the row with ti and remove the entry in the 
(i, l)-position (which is necessarily an ti) making it an empty box. If there is no 
entry in the ith row which is greater than h, then adjoin h to the end of the row. 

If & G Bi, then insert h into the jth column as follows: If there is an entry in 
the jth column which is greater than h, then displace the topmost such entry and 
insert h into its position. If there is no entry in the jth column which is greater 
than h, then adjoin h at the end of the column. 

(3) Set h equal to the displaced entry and change i to p + I and j to q + I where (p, q) 
was the position of the displaced entry. Repeat step (2) until an entry is adjoined 
to the end of a row or a column, or an empty box is created. 

(4) Let (a T)' be the result of steps (l)-(3). Set {a T) = {a T)' if (a T)' is 
a tableau, and {a ^ T) = jeu((a T)') if (a T)' is a punctured tableau. 

Lemma 5.4. Let T be an 5pO-tableau and assume a ^ B. Then (a T) is an 5pO-tableau. 

Proof. The algorithm for insertion of a letter into an spo-tableau is just Berele's insertion 
algorithm (see [Be, Lemma 4]) for the part of the tableau involving entries from Bq, and 
it amounts to the usual Robinson-Schensted algorithm (see [S, §3.3], for example) for the 
skew-tableau with entries from Bi . Consequently, the result (a T)' is either a tableau or 
a punctured tableau in AfW . From Lemma 5.2 it is clear that (a T) is an spo-tableau 
provided we identify a punctured tableau in S£ with the tableau obtained by removing its 
corner empty box. | 



Insertion of a word mto an spo-tahleau 
Recall that 

yVk is the set of words of length k in the alphabet B and 

Vk is the set of pairs (T, A) consisting of a spo(m, n)-tableau T of shape A and an 
up-down (r, n)-tableau A = (A*^, A^, . . . , A*^) of length k and shape A. 

For k >Q, define maps spOj,: Wk Vk inductively by 

(i) spOo(w) = (0, (0)), where w is the emptyword; 

(ii) If w = wi • • • Wfc is a word of length k and spo^_i{wi ■ ■ ■ Wk-i) = {T'''^ , (A*^, A"'^ , . . . , A*^ 
then spOj,(w) = (T^ (A°, A^, . . . , A*^)) where T'= = (wk T^-'^) and A*^ is the un- 
derlying partition of . 

Theorem 5.5. Tie map spOj, : Wk — ^ Vk is a bisection. 

Proof. It is clear that 5pOj.{w) G Vk for each w G Wk- To show that spOj, is a bijection, it 
is enough to find its inverse, and for that it suffices to find the inverse of the step {wi 
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rpt 1-j _ rpt ^ j^g^ yt 1 yt ^j^^ Underlying partitions of ^ and , respectively. We 
present an algorithm to produce wi from A^~^, and T^: 

(1) (initial step) If is obtained from A^~^ by adding a box, and if the entry in the 
corresponding box of is a, then delete the adjoined box from and apply (2) 
below beginning with h = a. 

If A^ is obtained from A^~^ by removing a box, then consider the punctured spo- 
tableau in S£ obtained by adding an empty box to in the corresponding corner. 
Apply mieu to obtain a punctured spo-tableau in MW . The empty box is now in 
position {i, 1) for some i. Fill the empty box with an ti, and then change the last 
ti in row i to an t* . Apply (2) below beginning with h = ti regarded as displaced 
from position {i, 1). 

(2) (general step) Suppose that an entry h is displaced from the (i, j)-position of T^. 

If & G 5o, then from the {i — l)st row of remove the rightmost entry which is 
smaller than h and put h in the empty box. 

If & G 5i, then from the (j — l)st column of remove the bottommost entry 
which is smaller than h and put h in the empty box. 

Repeat this step until an entry in Bq is displaced from the first row or an entry in 
Bi is displaced from the first column. That entry is wi. 

This gives the desired inverse process, and hence completes the proof. | 

We refer to the map spOj,: Wk ^ Vk as s'^po -insertion and the inverse described in the 

proof of Theorem 5.5 as s'^po- deletion. We illustrate spo-insertion and spo-deletion in the 
following example. 

Example. Suppose r = 4, n = 4, and k = Q. Then B = {ti < t^ < t2 < t"^ < ts < 
t4 < t"^ < vi < V2 < vs < V4}. Consider the word w = V2 t2 t^ ti vs ti of length 6. Then 
spo-insertion for the word w is given by 



ED' 





V2 







jeu 



V2 



V2 



t2 


V2 


V3 





t* 


V2 


t'J 





rj-ib 



h 


V2 




V3 



Thus spo(w) = (T, A) where T = and 



To recover w from (T, A) we apply spo-deletion: 



h 


U2 




V3 



h 


V2 


V3 





t2 


V2 


V3 





and We = 
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t2 


V2 


V3 





V2 



and W5 = vs, 



T4 



V2 



t2 


V2 






V2 






inj eu 


t2 





tl 


V2 


t2 





f* 

n 


V2 


t2 





and W4 = tl 



f* 
n 


V2 


t2 





t2 \ V2 \ and W3 =tl, 



T2 = \ t2 \ V2 



I V2 I and W2 =t2, 



= I V2 I and wi = V2. 
Thus, we get w = V2 t2 ti V3 ti. 

Corollary 5.6. Foi positive integers m = 2r and n, 

[k/2l 

(m + n)*^ = udA(r, n) • spo;^(m, n) 

h = \\-k-2h 

where udA(r, n) is the number of up-down (r, n)-tableaux of shape X, and s];>o^(m, n) is the 
number of spo(m, n)-tableaux of shape X. 



Proof. This is an immediate consequence of Theorem 5.5 by counting the number of elements 
in the sets Wk and Vk- I 



Remark 5.7. By ([We],[W]), the dimension of the irreducible module 5a, A G 5^ = {A h 
k — 2h I h = 0,1,..., [A;/2J}, for the Brauer algebra Bk(n — m) is the number of up- 
down tableaux of length k and shape A whenever |n — m| > k. This is also the number 
of up-down tableaux of length k and shape A' whenever |n — m| > k. By Theorem 4.24, 
char(T'^) = sc\(Z). Thus, Theorem 5.5 gives an explicit bijection realizing the identity 

T/®*^ = 0T^®5a' 

A 

from Proposition 4.2 (b). 
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